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ABSTRACT. This paper presents the development and testing of a Virtual 
Camera (VC) system to improve astronaut and mission operations situation 
awareness while exploring other planetary bodies. In this embodiment, the VC 
is implemented using a tablet-based computer system to navigate through inter- 
active database application. It is claimed that the advanced interaction media 
capability of the VC can improve situation awareness as the distribution of hu- 
man space exploration roles change in deep space exploration. The VC is being 
developed and tested for usability and capability to improve situation aware- 
ness. Work completed thus far as well as what is needed to complete the project 
will be described. Planned testing will also be described. 
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1 INTRODUCTION 

The Virtual Camera (VC) for human deep space exploration is a virtual assistant that 
is being developed for use by astronauts and other associated mission operations per- 
sonnel as they use human-piloted rovers to explore the surface of other planets [1]. 
The VC concept is based on incremental upgrading of a suboptimal 3-D geographical 
and geological database. It is an interactive window on the world as we know it at the 
time it is being used. Such an interactive window enables the user to maintain better 
situational awareness, to navigate and further explore space. It can be used for train- 
ing, data analysis and augmentation of actual surface exploration. 

The VC for human space exploration was originally described for use by astronauts 
navigating a surface exploration rover on a remote body such as the Moon [1]. Further 
analysis has indicated that such an interactive window to data in multiple dimensions 
has many other applications in domains such as aviation, medicine, control systems 
and finance. This paper presents a tablet version of the VC for deep space exploration 
using a scenario-based design approach. This work brings to human space exploration 
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the use of advanced interaction techniques not previously used in spacecraft cockpit 
systems. 

2 MOTIVATION 

There are significant challenges for astronauts while operating in the space environ- 
ment, including increased levels of stress and workload, combined with decreased 
situation awareness. Astronauts and their exploration are evolving in an environment 
that is characterized by progressive discovery. 

Human deep space missions will require increased roles for astronauts on-board as 
well as increased on-board automation. Previous work has determined that robots and 
humans need to work together in co-coordinated efforts [2]. It is envisioned that the 
VC can integrate data from both human-piloted and autonomous robotic explorers to 
improve the safe and efficient human exploration of a deep-space location (i.e., 
Moon, asteroid or Mars). On-board software and displays will integrate and manage 
all relevant systems as well as mission data. This information will include system 
health and status, caution and warning, traverse execution and mission timeline pa- 
rameters. 

There will be a need to make decisions based upon the collection and analysis of 
raw data to provide predictive information [3]. This information needs to be present- 
ed to crews in a way that enhances situation awareness. One way to do this is with 
new interactive environments such as 3-D tablet computer systems using advanced 
interaction techniques based upon accelerometer- and gesture-based inputs. 

3 RELATED WORK 

Interactive cockpits, context-sensitive systems and remote agent knowledge represen- 
tation are being developed for a number of domains including aviation [4], nuclear 
power plants [5] and even passenger car driving [6]. The VC will demonstrate these 
requirements in a system usable for deep space human exploration. 

In early tests using a human-piloted rover in space-simulated environments it be- 
came clear that astronaut situation awareness needed to be improved [7]. Often when 
operating in environments such as the NASA Desert Research And Technology Stud- 
ies (RATS) space exploration test bed, rover operators needs people outside the rover 
vehicle to guide for safe operation of the vehicle. This will not be possible for all 
rover or other space vehicle operations in deep space. 

4 DESIGN GOALS 

In deep space, situation awareness will be difficult due to the lack of input from the 
ground that astronauts are typically used to as well as the skill-retention concerns and 
environmental isolation from the environment under exploration. 


The VC is playing the role of a remote agent for mission operations personnel and 
scientists. Functions that were previously done by human experts on the Earth for 
missions near Earth will be transferred to the on-board VC. With this transfer there 
will be emergent behaviors, which will be explored and identified through this re- 
search. 

NASA is currently developing technology for use with deep space surface explora- 
tion. A centerpiece of this exploration effort is the Multi-Mission Space Exploration 
Vehicle (MMSEV). This vehicle is being designed for use by astronauts on a variety 
of missions including Near-Earth Asteroid exploration. It is anticipated that the VC 
would support exploration conducted by this vehicle. Figure 1 shows a MMSEV con- 
cept that is based upon the Surface Exploration Vehicle that has been tested at the 
NASA Desert RATS analog test-bed. Part of the goal of designing a VC for space 
exploration is to afford the most natural and intuitive human-system interaction. A 
tablet computer interface provides a small, portable platform with a very common 
user interface that is gaining wide acceptance. 



Fig. 1 . Multi-Mission Space Exploration Vehicle (MMSEV) for Astronaut Deep Space Explo- 
ration (NASA). 

A tablet allows gestures rotational interaction that astronauts have expressed that 
would be useful in the space environment. Two tablets are being used by Russian 
cosmonauts for recreation on the International Space Station [8], and by pilots for the 
use of electronic charts in commercial aircraft cockpits [9]. 


5 WORK COMPLETED 


A human-centered design approach is being used to develop the tablet-based VC. 
The first step was to develop low-level prototypes to capture the basic design re- 
quirements for the VC. These were then presented to potential users during the fall of 
2011 at the NASA Desert RATS test sessions. Horizontal prototypes were developed 
that captured the basic interaction requirements for the VC and were used to demon- 
strate an operational scenario. Currently, a complete vertical prototype is being devel- 
oped with all of the interaction capability using a tablet PC device. 

5.1 User Requirements Elicitation 

The authors conducted a survey of potential users and stakeholders (astronauts, 
mission operations personnel, and scientists) for the VC system at the 2011 NASA 
Desert RATS analog exploration testbed. The main questions during potential user 
interviews involved their background, general system uses, interface type and data 
display parameter formats desired. 

User feedback indicated a device that provides real-time sensor knowledge display 
combined with multi-modal information display (vision and auditory for instance) 
would be useful. They also suggested that the VC should help in the navigation pro- 
cess and also display information about consumables such as power level, fuel levels 
and life support systems. This information was used to develop initial horizontal pro- 
totypes showing possible uses, screens and interactions for the VC tablet device. 

Use cases for the VC were defined, and then a horizontal prototype was devel- 
oped. This allows human-in-the-loop simulations to take place where the system can 
be tested with a variety of users. Other scenarios can also be illustrated in this way. 
Users can interact with actual system prototypes, which are being developed on the 
tablet-based computer system that will be used for the end product. This then allows 
the full functionality to be incorporated after initial user testing and feedback. 

The horizontal prototype screens show the functionality and capabilities of the VC 
although the database interaction was not yet be possible. This horizontal prototyping 
allows astronaut, scientist and mission operations personnel to provide feedback as to 
the usefulness of the screens being developed. 

Asteroid Itokawa has been modeled using a stereo lithography database format 
(STL) created by Gaskell [10] based on data from the Japanese asteroid mission 
Hayabusa. This data was used in the developed prototypes to simulate flying close to 
asteroid Itokawa and the resulting VC interactions. 

The VC interface screen in Figure 2 shows the actual asteroid surface as projected 
by the STL file. This is how an astronaut would interface with the VC as they ap- 
proached the asteroid in an MMSEV. The vehicle’s velocity in X, Y and Z coordinat- 
ed is shown in the upper left. Next to the actual velocity are bars of relative velocity. 
The color green represents a safe speed for the operating environment. If the velocity 
was slightly high, then the bar would turn yellow and if it was a dangerous level it 
would turn red. The velocity vector is simulated by the green arrow, which also points 
in the basic direction of travel. 



Fig. 2. The VC Horizontal Prototype Interface. 


An information cartouche in the lower right shows icons for communication link, 
thruster systems in use, electrical power and collision potential going clockwise from 
the upper left. Other information elements include a radiation sensor-warning icon in 
the lower middle indicating an on-board radiation sensor has detected radiation. In the 
lower left the blue beaker icon indicates an area of scientific interest. 

In this example scenario the astronaut can determine to fly toward the beaker icon to 
attain a mission science goal or ignore the icon and fly in another direction. If another 
direction is chosen, other areas of interest could be displayed by icons. Sensor data is 
also compared to the current vehicle configuration and operational mode to determine 
possible safety concerns and out of nominal operation. 

Figure 3 shows some of the safety information displays possible in the VC. In this 
case the astronaut is flying closer to the asteroid and is close to a mountain feature on 
the surface. Dust may be kicked up as he approaches and a dust warning has been 
triggered. The fuel level in the thruster triad currently active has reached a threshold 
that has triggered the rocket icon to turn red. Also the communications link with ei- 
ther the Earth or a nearby habitation system has deteriorated causing the communica- 
tions (antenna) icon to turn yellow. 

As the vehicle approaches the area of interest (in either a simulation mode or later in 
actual deep space exploration) the beaker icon changes to surface colors to indicate 
the exact area for exploration. This allows the astronaut to determine where to aim 


high resolution sensors, take high resolution images or to collect samples. Similarly 
mission operations personnel or scientists could “fly” mission scenarios and deter- 
mine plans for exploration sorties at the actual asteroid. Annotations and other infor- 
mation screens can also be displayed as required when the exploration context calls 
for them. Examples could be radiation sensor alarms, loss of communication, and 
impending contact with the asteroid surface or a crater on the moon. Colors could 
represent either physical concentrations or an exploration priority scheme. 



Fig. 3. VC Prototype Screen Showing Warnings 


These screens can be put together into a storyboard as the astronaut in the simulated 
MMSEV flies closer to and around the asteroid surface. This allows a movie to be 
created that simulates a mission scenario. Other types of information displays possi- 
bilities with the VC interaction include coloring the surface to correspond with certain 
areas of scientific or exploration interest. The elevation information would still be 
visible but additional information about areas that may potentially be explored could 
also be represented by different colors. 

Emergent uses of the system and behaviors enabled or limited by the VC are dis- 
covered by the use of prototypes. A vertical prototype with the interaction function- 
ality afforded by a tablet computer is now under development. This will allow com- 
plete testing of interactivity of the system. Figure 5 shows a screen from the VC verti- 
cal prototype using the Google Maps database. The information cartouche is dis- 
played with the collision icon active if the user zooms in beyond a settable level. The 
direction arrow shows the direction of the accelerometer interaction sensor. The upper 
left shows where the VC application started as a central empty blue circle (in terms of 
geographic position) as well as a second red dot indicating the current view position 


in relation. Scale is also user settable. Scientific (beaker), resource (water drop) and 
user annotations (information) icons are also present. 

6 FUTURE WORK 

The final product of this work will be a tablet-based prototype VC device using the 
unique interaction capabilities of the tablet platform. These include portability, accel- 
erometer-based 3-D interaction and the ability to be submerged into the 3-D environ- 
ment. The interaction features such as icons, colors and annotations described are 
being incorporated. 



Fig. 4. The VC Vertical Prototype Interface Using Google Maps as the Database. 


This VC device will form the basis for evaluating the utility of the interactive data- 
base concept for improving situation awareness in the unique environment of deep- 
space human-piloted space exploration. Both mission planners and astronauts can 
conduct a simulated traverse using the VC, annotating areas of exploration interest or 
safety concern prior to conducting the actual traverse in the field. The tablet GPS 
location system can then be used to assist the astronauts as they follow through the 
traverse at a terrestrial space exploration analog site such as NASA Desert RATS. 


7 CONCLUSION 

The virtual camera for planetary exploration will assist in future exploration involving 
humans and robots. It is designed to provide improved situational awareness and 


augmented reality to define areas of interest and safety concern on remote planetary 
bodies. It will act as remote agents for mission planners and scientists capturing their 
knowledge and expertise as astronauts explore the solar system. It is claimed that the 
advanced interaction media capability of the VC can improve situation awareness as 
the distribution of human space exploration roles change in deep space exploration. 

Human-centered-design techniques are being applied to the development of this tool 
involving all potential user groups from the beginning of the design process. Moving 
forward, the VC prototype will be finalized and further testing with domain experts 
will take place. Changes found in this testing will be incorporated into an updated VC 
and final evaluation will take place comparing this system to other interaction devices 
currently being developed. Recommendations will be made for future work to create a 
flight-ready VC capability. The advanced interaction possibilities with the VC will 
allow new types of displays and an immersive environment to assist astronauts to 
comprehend and project into the future situation awareness as their autonomy from 
ground control resources increases in deep space. 
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